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INTRODUCTION 



Evaluators— trainers, psychologists, management consultants and professors— have been systematically 
evaluatmg management training programs for over twenty years. And, also for over twenty years, eval- 
uators have been publishing articles about their findings. Consequently, a wealth of experience has 
accumulated from which anyone interested in evaluation can profit. 

You may be interested in the evaluation of management training because you are a trainer and want to 
evaluate courses yourself. You may need to know more about what constitutes good evaluation design 
Perhaps you a.re a manager who must make decisions that are based on evaluations done by others 
You may need help in judging evaluations. Whatever the reason for your interest, you may not have 
the time to read all or even a large share of the articles that two decades of steady work has produced 
You need a way of judiciously sampling the field. 

We have provided that way by abstracting twenty-eight representative articles. In making our choices 
we looked at evaluations of training that varied according to the organizational setting in which the 
traming was given, according to the managerial functions that were being taught, according to the top- 
ics that were covered and accc-ding to the instrucJional strategies that were employed. 

The table on page iv shows what kind of design was used by each evaluator whose work we have ab- 
stracted. Using the table will make it easier for you to find the kind of evaluation that you are interest- 
ed m. It will also make it easier for you to compare the design of one evaluation with that of another. 

TTiis publication includes a list of selected references and^ a glossary of evaluation terms. Each term 
that is defined in the glossary is printed in boldface the first time that it appears in each abstract. The 
only exception to this rule is the term "evaluation," which is not in boldface because it appears as a 
headinc >r< each abstract. 
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CHARACTERISTieS OF EVALUATION DESIGN 



The table that follows on page iv summarizes each of the evaluations that we have abstracted. It 
shows the characteristics of the design that each evaluator used to gather the mformation and the types 
of information that (s)he gathered. (The evaluations are listed chronologically so as to yield a historical 
perspective and to reveal any trends that may have developed in the last twenty years). 
The design characteristics of an evaluation dictate the strategy that the evaluator uses to collect data, 
bid (s)he use a control group as well as an experimental group? Were the participants picked at random 
or by matching? Were they given a pretest, a posttest or both? The table answers each of these ques- 
tions for each of the evaluations. 

An experimental group is the collection of subjects who receive the training. A control group is a 
collection of individuals who are similar to the subjects in all relevant ways except one: they do not 
receive the training. An evaluator can use a control group in determining whether the training has af- 
fected the experimental group. (S)he tests both groups and compares the results. Presumably any 
change shown by the experimental group that is not shown by the control group is the result of the 
training. 

The members of the control group should be as much like the members of the experimental group as is 
possible Otherwise it would not be reasonable to determine from a comparison of the two groups what 
the members of the experimental group have gained from the training. For example, suppose that the 
members of a control group had each worked with the Federal Personnel Manual for ten years. Sup- 
pose further that the members of the experimental group had never seen the Manual and that their 
training consists of a one-week introduction to it. If both groups are tested when the f Peridental 
group has completed its training, the experimental group will probably show less knowledge of the 
Manual than the control group will. Clearly, however, it would be a mistake to deduce that the experi- 
mental group has not "profited from the training. 

To ensure that the members of the control group are similar to the members of the experimental group, 
the members of both groups are usually picked either by matching or by random selection. 
To match is to deliberately pick peopl^for the control group whose backgrounds arelhe same as those 
of the members of the experimental gr.oup. Background, for this purpose, can be defined as those fac- 
tors that might influence the experiment: age, status, education, etc. To pick by random selection is to 
choose participants in such a way that all people who are eligible to be participants have an equal 
chance of being included. Furthermore, participants must be chosen in a way that ensures that the 
selection of one participant has no influence on the selection of any other. 

If a control group is not used, it is particularly important that the members of the ekperimental group 
be picked at random. To understand why, consider how valid an evaluation would be if it were an 
evaluation of a course that was designed to teach conference leadership and if the course were given 
to an experimental group composed of participants who had each successfully led several conferences. 
As for the characteristic that has to do with pretests and posttests, if an evaluator has given both to 
the subjects, (s)he has given the subjects the same test twice; once before they start the training, and 
once after they complete the training. By comparing the results of the pretest to those of the posttest, 
the evaluator is able to determine whether the subjects have changed during the time between the 
tests. Changes are presumably the result of the training. 

The presence or absence of all of these characteristics influence the accuracy of the evaluation find- 
ings After all, if an evaluation is notWell designed, it is possible to ascribe effects to training that are 
actually caused by other factors. Unfortunately, designing and using an evaluation strategy that mcor- 
ixjrates all of these characteristics involves time, money, and planning. Therefore, the designs of many 
of the evaluations that we have abstracted do not include all of these elements. 
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The table on page iv also shows what kinds of information were collected in ^^^^h evaluation. The data 
is categorized on the table according to the system suggested by Donald Kirkp^t"^!^'* thai is, under 
the following headings: reaction, learnii^, behavior, and results. Reaction and learning measured 
while the participants are being trained. Reaction refers to whether the participants like the training, 
including the materials, the instructors, the facilities, the methodology, the me^'^ the content. 
Learning refers to the facts, skills, and attitudes that the participants gain froni tl^" training. Behavior 
and results are measured after the trainees return to their jobs. Behavior means if^^ trainee's perform- c 
ance on the job, and results refers to the impact that his or her training has on the ^^^^^'Zation. 



'Kirkpatrick, Donald L. (ED.). Evaluating Training Programs. Madison. Wisconsin: American Soqic^V ^raiining and Devel- 
opment, Inc., 1975. 
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TABLE A: EVALUATION STUPY PESIGN CHARACTERISTICS 

^measurement 



Source/ Year 


page 


Control 
Group 




25 . 


Yes 


Blocker n95*^^ 


7 


No 


Qooda^rp (1955) 


13 


Yes 


^fOQ^n&Harriton 


.'33 


Yes 




42 


No 


BUCK'S o 
_^[^*it8tter (1959) 


8 


Yes 


(19607^y- 


23 


Yes 


HillrTian (1962) 


17 


No 


^Qgdaf^rfi (1963) 


14 


Yes 


Blake et al 
(1964) 


5 


No 


(1964) 


32 


V Yes 


— ___j^e|ianu (lyDo; 


28 


2 


\ 1 ^yj*^ ) 


30 


Yes 


(1966)* " 


-6 


No 




1 


. Yes 


SchwarT et 
(•%8) 


40 . 


Yes 


ValiQ.,^» MQf^Q\ 


40 


Yes 


^'Ison et al 


45 


Yes 


Thoru^ (1969) 


39 


No 


^<^Mn (1971)' 


35 


No 


& Slocum 

_(1975) 


. 15 


Yes 


.^^oyne (1973), 


9 


Yes 


Mo^^tein. et al ' 


1 1 


No 


J^jjjuiies (1973) 


27 


Yes 


f shton & Gibbon 
\'974) 


3 


No 


"•^^^n^^ch (1974) 


19 


Yes 


H^riS^ker & Hall 
0974) 


21 


No 


^^x^ey.& Nemeroff 


43 


Yes 




— ■65 v-Mi— -. ..-cifically address c3«^9°''y- 

^'^ccielland reports three studies, ong of wtiicti tias used a m^^'^*^^° control g^^^p. 
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Albanese, Robert. A case study of executive development. Training and Development Joiinal, 1967, " 
21(1), 28-34. ^ ^ 

Albanese, a member of the Obh State University faculty^ evaluated the long term effects of an ongo- 
ing e;:ecutive development program run by the National Restaurant Association (NRA), His evaluation 
demonstrates how the use of a control group can increase the accuracy of the evaluation results, 



TRAINING 

• The one-week course, given in s^even locations 
" in the U.S., covered basic management con- 

- A cepts, iniprovemept of managerial attitudes and 
behavior, supervisory functions of the manager 
and standards for restaurant operation. Lec-i 
ttires, informal group discussions, case studies, 
role playing, question and answer sessions, as 
Well as consultation with faculty members con- 
veyed course content. The trainees in the ses- 
sion which Albanese evaluated, were eleven 
Women and sixty-pne men v/ho managed restau- 
rants of various types and sizes. 

EVALUATION^ 

Albanese began his evaluation in 1961, three 
years after the inception of the NRA's manage- 
ment course. He selected a control group from 
the NRA membership catalog. Statistical com- 
parison of background factors showed that the 
groups did not differ significantly,' and could 
therefore be compared on evaluatiop measures. 
Prior to training, managers in both the experi- 
»n^tal and control groups received five sets of 
questionnaires with, instructions, one to be filled 
out by the manager and four for his or her sub- 
ordinates. Subordinates completed the Leadership 



Description Behavior Questionnaire (LDBQ), a 
•measure of subordinate perception of leadership 
behavioral style. Managers filled out the Respon- 
sibility Authority Delegation (RAD) scale which 
Pleasures these elements for individuals who 
occupy managerial jpositions. Only question- 
naires returned before the end of the session 
were included in the analysis. 
Finally, six weeks, after training, each respon- 
dent received a second questionnaire with his or 
'^er name typed on it directly. This insured that 
the designated person received and completed 
:.the questionnaire and that the questionnaire 
could be identified upon its return. Question- 
naires returned after the five-week cut^-off point 
were not included in the analysis. 

FINDINGS 

Managers in the experimental group improved in 
their perceived ability to take responsibility and 
delegate authority. Their subordinates also 
perceived an improvementjn managerial use of 
authority. These improvements,' however, were 
not significantly different from parallel improve-* 
, ments in the control group. Because both groups 
had an equivalent amount of change, it. was 
impossible to attribute positive effects directly to 
the training. 
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Ashton, David and Gibbon, Brian. Evaluating a training program in the probation service European 
Training, 1974,3(1), 62-74. ^ , 

Ashton and Gibbon apply a mix of formative and summative evaluation techniques to the development 
of a management course for probation officers. 

'^''^AINING ness School for Probation^ Officers in the Dur- 

Beginning in Feburary 1973, an experimental ham Probation Service. It was the first time that 
course was run by the Durham University Busi- the university had taught management concepts 
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to a social service department This type of 
training within the Probatior Service had also 
previously Been limited, 

''The course objective was to impart an under- 
standing of management concepts a;id their ap- 
plication whhir the probation service context. 
Subject matter was presented by four m^^mbers 
of the university staff op the university premises 
in five one-day sessions, each three to four 
weeks apart. 

EVALpATIOi^ 

Because this was the first course of 'its kiqd 
conducted by the universily, the evaluators se- 
lected ,a strategy which contained elements of 
formative, evaluation. They continually gathef^d 
information during the development and execu- 
tion of the couxse and fed it back to course de- 
veloper^ and instructors. In this way evaluators 
were able to achieve the ro istant refinement of 
the course as it evolved. 

Prior to training, members of the teaching staff 
made several visits to the Probation Service 
Office to look into the organization's problems. 
In addition, prospective participants 'were sent a 
precourse questionnaire asking for their own cri- 
teria of course success. The responses were ana- 
lyzed and presented to the teaching i^^aff before 
the course began. The evaluation T function 
helped in formulating teaching policy at the out- 
set of the program by producing relevant iiifor- 
mation which changed assumptions made in the 
early stages of the course's development. 
While the course^ was in session, two instru- 
ments were used to gauge participants' percep-. 
tions. Session rating scales provided an indica- 
tion of participants', reactions to the key dime;i- 
sions of the course. A session assessment form 
asked open-ended questions about the content of 
each session and its perceived benefit to the par- 



ticipant. These comments were t3ped out and 
presented to the instructors shortly after the end 
of each jsession. The, information v;as also fed 
back to the trainees in* the hope that they would 
find each others' comnients valuable and would 
continue to'fill out the forms. 
THe evaluators, the teaching staff, and the Pro- 
bation ServiceiiJOfficer jointly formulated the de- 
sign for a final questionnaire. Hie main reason 
for this survey was to obtain feedback on the 
overall effects of the course, indicating areas of 
weakness which could be strengthened for sub- 
sequent offerings. The questionnaire, sent out 
three months after the course had ended, elicited 
primarily personal perceptions from which the 
value of the course to the organization was in- 
ferred. According to Ashton dx\d Gibbon, "At 
first glance this questionnaire woVld appear to be 
a straight forwaid 'reactions' Evaluation tool, 
but it must be. remembered th^ respondents 
were the to^ manageirient strata of the service^ 
and hence, value to them personally must be 
aligned with value to the organization." 

FADINGS 

Overall, the evaluators found that the informa- 
tion provided by the evaluation effort was of 
dual benefit. It served to improve the training 
program during its development, as well as help- 
ing to assess^ the value of the training after- 
wards. The evaluation was especially useful in 
designating instructional goals, establishing rele- 
vant course jcontent and providing the necessary 
feedback on course effectiveness. 
Through the application of formative evaluation 
techniques the evaluatoi, university staff ar^d 
Probation Officer were able to combine their tal«- 
ents and develop a course which addressed the 
needs and the interests of their audience. 



Blake, Robert, Mounton, Jane S., Barnes, Louis B., and Greiner, Larry. Breakthrough in organization 
development. Harvard Business Review, 1964, 42(6), 133 

External consultants, Barnes and Greiner, used a combination of quantitative and qualitative measures 
to d^rmine the effect of Managerial Grid Training upon or^nizaticnRi effectiveness. 

TRAINING Managerial Grid Training, a one-week lab-semi- 

Eight hundred managers in a 4,000 member divi- nar using problem solving and group discussion 
sion of a large petroleum company received techniques and analysis of self asse^^smerit ques- 
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tionnaircs. Exercises conc-.^ntral :d on interper- 
sonal relations and setti :r and ? oiev ng goals. 
Senior line managers servf i as inslructors. 

EVALUATION 

The evaluation addressed three main objectives: 
(D to determine if training in behavioral science 
concepts could indua- organiratiorai change 
through the behavior or the trained managers; 
(2) to determine if introspective analysis dam- 
ages the mental h^th of trainees; and (3) to 
assess changes in individual and group behavior. 
To assess the degree of attainment of the stated 
objectives, the e valuators used questionnaires, 
interviews, and observations. Past and present 
company records were examined in order to 
separate program from rionprogram effects. The 
data were collected over a four month period 
about one year after the first course offering, 
while the last offering was in progress. The e val- 
uators examined productivity and profit indexes 
for the periods before and after training. Post- 
training surveys provided . information on 
changes in opinions and attitudes, and inter- 



views and conversation yielded evidence of be- 
havioral changes. 

FINDINGS 

During the year that the training program was in 
operation, the company reported a considerable 
rise in profits and a decline in costs which they 
claimed were not attributable to price changes in 
raw materials and finished products or reduc- 
tions in the labor force. Overall, employee pro- 
ductivity increased without additional investment 
in plant equipment. Other positive effects de- 
monstrated by the varied measures included a 
catalog of organizational problems solved during 
the year, increased frequency of - meetings, a 
change in criteria for management appraisals and 
a shift towards the 9.9 or high production and 
high people concern as measured on Blake's 
Managerial Grid. This study demonstrated that 
behavioral science concepts can b^* incorporated 
into individual behavior, leading to. more effec- ^ 
tive group and, utiimately, organizatio'nal func- 
tioning, without any adverse effects on employee 
mental health. 



Blake, Robert R., and Mouton, Jane S. Some effects of managerial grid seminar training on union and 
management attitudes towards supervision. Journal of Applied Behavioral Science, 1966, 2(4), 387-400. 

Blake and Mouton evaluated the effect of Managerial Grid training on both the production centered 
orientation of managers and the employee centered orientation of union leaders, rs^ 



TRAINING 

Identical but separstv Managerial ..Grid 'Seminars 
were conducted tor thirty-three management 
personnel and for twenty-three union officials, 
all of whom had management or staff responsi- 
bilities within the local. The training experience, 
relying mainly upon a laboratory format, also 
featured pre-course readings, organizational and . 
self-assessment instruments and structured 
learning experiences. The purpose of the Semi- 
nars was to aid management personnel and o 
union officials in gaining a fundamental under- 
standing of modern behavioral science concepts 
concerning sound people-production relation- 
ships. 



EVALUATION 

Measurement for the evaluation was taken both 
before and after training. To assess both groips' 
attitudinal changes toward supervisory practices 
and consequent changes in managerial style, the 
evaluators used a measure derived from the 
goals of the Managerial Grid. The forty-item 
forced choice questionnaire assessed beliefs 
concerning an individual's supervisory practices 
regarding the integration of people in a produc- 
tion setting. Five distinct managerial styles were 
noted by the grid approach and measured by the 
questionnaire: maximum concern for both peo- 
ple and production (9,9), minimum concern for 
both production and people (1,1), maximum 
/ 

■/ . 
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concern for produttion and minimum for people 
(9,1), minimum concern for production and max- 
imum concern for people (1,9), and a balanced, 
but moderate, concern for both (5,5). 

FINDINGS 

Initially, managers scored higher than union rep- 
resentatives on the sxyzs ha^jdng a high produc- 



tion orientation and lower on those with a low* 
people orientation. No differences between 
groups were found on the moderate style. After 
Managerial Grid training, both groups showed 
increased concern for people, but this increase 
was more marked for the managers. 



Blocker, Clyde E. Evaluation of a human relations training course. Journal of the American Society of 
Training Directors, 1955. 9(3), 7-8, 46. ^ 

Blocker, a university professor, demonstrates the use of previously collected or historical data to as- 
sess the results of a human relations training program. , 



TRAINING 

This study addresses a leadership training pro- 
gram conducted for seventy-eight supervisors 
from all levels of management of an insurance 
company. The purpose of the course^ was to help 
supervisors understand their personnel responsi- 
bilities and give them some techniques which 
would make their leadership more positive and 
effective. During ten two-hour sessions spanning 
an eiRht-month period, trainees covered supervi- 
so: responsibilities, leadership, "motivation- 
emploj'ee cooperation, employee attitudes .an»^ 
their effect upon production, counselling, giving 
orders and discipline. Instructional strategies in- 
cluded selected readings, films, slides, group 
discussions and role playing. 

EVALUATION 

The evaluation, which took place after the com- 
pletion of the training program, overcame some ' 
of the problems of using only posttraining mea- 
surements by using historical data. The evaluator 
selected fifteen of the seventy-eight participants 
at random to lake part in the study. Then, based 
on observation of on-the-job behavior taken dur- 
ing the preceding two years, the selected partici- 
pants were classified into two groups according 
to their style of leadership; eight were classified 
as democratic and seven as authoritarian. 



Three montl^s after training, the evaluator exam- 
ined standard printed interview forms to deter- 
mine if members of the sample had experienced 
behavioral change. The organization requiied 
supervisors to fill out these forms when he/she 
interviews an employee for any reason. These, 
forms contain the reason for the interview, the 
attitude of the employee, supervisor's comments 
and action taken. All forms were classified ac- 
cording to the tone of the interview: threatening 
interviews were classified as authoritarian; re- 
warding and cgnstructive ones were seen as 
democratic. To ''determine whether any of the 
fifteen supervisors had changed their behavioral 
patterns. Blocker compared the pretraining clas- 
sifications with those derived from the posttrain- 
ing examination of interview: forms. Seventy 
percent of . the previously authoritarian group, 
responded in an authoritarian fashion and eighty 
percent of the democratic groups respotided' 
democratically. " . . .1 

^ ' - . . 

FINDINGS 

Blocker concluded that the course was iJi\able to 
overcome natural resistance to change found in 
supervisors. He stressed the importance of top 
management support of new policy, and the 
importance of training on an individual basis. 
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Buchanan, Paul G., and Brunstetter, Phillip H. A research approach to management improvement: Part 
1, Part li. Journal of the American Society for Training Directors, 1959, 13(1), 9-8, and 13(2), 18-27. 



In-House investigators, Buchanan and Brunstetter, evaluated a T-group training program to determine 
its effects upon organizational climate and individual interpersonal behavior. 



TRAINING 

The three day, two night T-group training pro- 
gram took place at an inn two hours from the 
work site. Two hundred twenty-four managers 
from one large department of the Republic Avia- 
tion Corp.. attended. 

The training had two main goals: improving or- 
ganizational climate with respect to the reward 
system, role Itlarity, morale, inter-level commu- 
nication and employee utilization; and develop- 
in^tndividual interpersonal problem solving and 
behavioral skills* Instructors conveyed course 
content through small group (9-12) lab-type con- 
ferences^ supplemented by larger group (12-24) 
factual lectures. 

EVALUATION 

Three to seven; months after the managers had 
been trailed, they received a questionnaire ask- 



ing them for a comparison of how things were 
currently done within their work unit in relation 
to how they had been done a year ago. The 
questionnaire, directed at organizational devel- 
opment, required managers to rate changes in 
the effectiveness of various organizational func- 
tions. A second group of 133 untrained managers 
from another departmental unit, acting as a con- 
trol group, also completed the questionnaire. 

FINDINGS 

The researchers determined which organizational 
functions were controlled by management and 
compared the two groups along these dimen- 
sions. The trained group reported a greater 
number of positive changes. 



Burgoyne,^'Jol 



n G. An action research experiment in the evaluation of . a management development 



<ourse. Journdjl of Management Studies, 1973, 10(1), 8-14. 



Burgoyne tested an action research design in the evaluation of a vniversity-based management training 
^program. This design permits some^ of the advantages of formative evaluation to tiejtransmitted to an 
otherwise summatiye evaluation study. 



TRAINING 

Burgoyne evaluated a twelve- week management 
coOrse offeired . by the Manchester Business 
ScbfetJt ir\ England and taught by the school's 
faculty. 



evXluation 

In this study^ Burgoyne demonstrates the appli- 
cability of an "action research" approach to the^ 
evaluation of a management training program. 
'The traditional researcher observes and' analyzes 
the research situation. The actiop-researcher, on 
the fither hand, uses the j:e^earch findings to 
bring about change in the research situation gfrid 



monitors the change brought aboui^by this inter- 
'Vention, which is what Burgoyne'did. 
The specific action reseacch design employed by 
Burgoyne is called^a^^ested" action research 
design, in whictuthe researcher is actually carry- 
ing out twgin^ependent studies, one of which is 
"inside^the other. First the researcher observes 
. the^esd^arch situation and uses his or her obser- 
.-^vations to influence the situation ("influencing 
procedure"). In the second phase the ''monitor- 
ing procedure^" the researcher'assesses the con- 
sequences^of change brought about through the 
intervention. This-second study is actually taking 
place within the framework provided by the first 
one. . 



si 



During the first or "influencing" phase of his 
research Burgoyne chose to study twelve of the 
thirty-six predefined course segments. He inter- 
viewed the faculty members responsible for 
these segments and used the resulting informa- 
tion to develop instruments which allowed these 
segments to be compared to future offerings. 
During the interviews, faculty members desig- 
nated objectives and criterion measures and pro- 
jected possible behavior changes which could be 
expected if their course segment was successful. 
About half of these instruments included objec- 
tive tests. The instruments were administered 
both before and after training. The posttraining 
measurement asked, in addition, if participants 
expected subject matter to influence their work, 
and if so, would it lead to a demonstrable effect 
in the organization. Information was fed back to 
instructors after the session was over for use in 
the preparation of the next course offering. 
The second or "monitoring" phase of the study 
tested the following ^Jiypothesis: The impact of 



the ''influencing" research will manifest itself in 
a greater increase in average participant rating of 
learning for those parts of the course that had 
been studied than for those parts which had not. 
When the university repeated the course, nine of 
the stvdied course segments and twelve of the 
unstudied course segments were taught by the 
instructor who had taught them previously. Parti- 
cipants in both the studied and unstudied groups 
rated perceived change in learning on a seven- 
point scale after they had completed the sieg- 
ment. , . 

FINDINGS 

A comparison made between the two groups 
showed that participants in the studied group 
reported a significantly greater amount of per- 
ceived learning, thus proving^ the stated hypoth- 
esis^ The success of this evaluation indicates 
that the action research method has great poten- 
tial for the improvement of ongoing courses- 



Goldstein, Stanley, Gorman, James, and Smith, Blanchard B. A partnership in evaluatio^. Training and 
Development Journal, 1973 , 27(4), 10-14. ^ ■ . . 

While the ""one sbot case study"" (or single posttraining measurement without control group) is general- 
ly considered the least acceptable approach to training evaluation, there are some times when financial 
organizatiohal and time constraints prevent the use of more rigorous designs. Goldstein et. al. demon- 
stratey how under such conditions, evaluators can .capitalize on available resources. 



TRAINING 

In conjunction with a Kepner-Tregoe research' 
associate^ a training center director and employ- 
ee development specialist undertook a study to 
determine whether ^ Kepner-Tregoe courses had 
been effective in meeting the. needs of the 
NASA Manned Space Craft Center in Texas. 
Over a five year period, 300 Manned Space 
Craft managers attended a total of sixteen offer- 
ings of the problem-solving and decision jnaking 
training course. 



EVALUATION 

The objectives of the evaluation were to assess 
course effectiveness, to determine if NASA's 



investment in the course had paid off in better 
problem solving, and to find out what levels and 
types "^of supervisors would benefit from the 
co?irse. To gain the answer to these questions, 
the evaluation team .designed a self report ques- 
tionnaire. After review by NASA executives and 
pilot testing had been- completed, the question- 
naire was sent to all 250 former course partici- 
pants remaining with NASA at the time of the 
evaluation. The , questionnaire included a, back- 
ground sheet, questions emphasizing changesUn 
behavior and job performance, and an optional 
sheet for noting specific incidente-^hen course 
concepts had been applied. In addition, the eval- 
uators obtained a measure of results by estimat- 
ing the return on investment attributable to the 
training program. 
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nNDINGS 

The results of the evaluation were encouraging. 
Program effectiveness was judged by two crite- 
ria: Perceived value and extent of application to 
the job situation. One half of the respondents 
reported the course as having significant value. 
Forty percent reported specific areas of benefit. 
More than one half reported frequent informal 
use of course concepts and slightly fewer than 
half reported repeated fprmal applications to 
major job concerns. 

In order to get an approximation of return on 
NASA's investment in the training program, the 
evaluators compared the cost of the course to 
the typical manager's annual salary. The course 
costs, including participant salaries, facility and 
program expenses equalled about five percent of 
a manager's annual salary. By contrasting this 
investment with payoff, in terms of perceived 
growth and performance efficiency attributable 



to the course, the evaluators estimated that the 
return on investment had been 200 percent the 
first year. 

Analysis of questionnaire data failed to pinpoint 
a single factor designating those levels and types 
of supervisors most likely to succeed in applying 
course concepts to job concerns. The data did, 
howeyer, suggest that "the more talented em- 
ployees received greater course benefit." In ad- 
dition, the application of course techniques to 
the job depended upon the kind arid level of po- 
sition which. the supervisor held and the degree 
to which his superior , and peers supported the 
use of these new techniqiies. 

The evaluators concluded with a. discussion of 
how course value can be maximized. Supportive 
organizational factors, application of knowledge 
on the job, and training employees at a live-in 
rather than an on-site facility all had an impact 
on maximimizing course value. 



Goodacre, Daniel M. HI. Experimental evaluation of training. . Journal of Personnel Administration and 
Industrial Relations, 1955, 2(4), 143-149 . ' 

In this 1955 article, Goodacre expresses the need for formal evaluation of training by training direc- 
tors, then demonstrates how to conduct an experimental training evaluation.^ 



The subjects .of Goodacre's experimental study 
were 800 individuals holding diverse managerial 
and supervisory, jobs within the B.F. Goodrich 
Company. All subjects had been designated as 
eligible for training. Subjects were randomly di- 
vided into two groups. One gfoup received train- 
ing; the second, served as a control. 
The trained group received eighteen ninety-min- 
ute sessions designed to improve their knowl- 
edge and skills in understanding human behav- 
ior, decision making, employee selection, and 
job evaluation. 'iVhile the conference method 
was the dominant instructional method used, 
some lectures and discussions were included. 

EVALUATION 

The ev^luator measured both groups before and 
after training. He used a^ttitudinal scales to mea- 
sure the managers* confidence^in the application 
of subject matter areas, their attitudes toward 



the company, and' job satisfaction. Achievement 
tests measured learning in subject matter areas, 
and supervisory ratings measured behavior at- 
tributable to course and overall job perforrh- 
ance. . ' \ 

FINDINGS 

In general, the control group did not change on 
any of the factors measured. The trained group, 
however, showed improvements in self confi- 
dence and learning. The trained ^oup also re- 
ceived slightly higher posttralning supervisory 
performance ratings than the control group, but 
this difference was not as^ pronounced as the 
changes found by the measures of learning. 
Goodacre concluded by stressing the need for 
data collection strategies ''meeting the condi- 
tions of having . an adequate criterion, controls, 
statistical analysis and a built-in experimental 
design" in order to "provide manaigement with 
facts as to the return on its training investment 
and aid in further program improvement." 
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Goodacre. Daniel M. III. Stimulating man management. Personnei Psychoiogy, 1963, 16(2), 133-143. 



Goodacre shows how several control groups can be employed to test simultaneously the relative effec- 
tiveness of several variations of a new training program. 



TRAINING 

Goodacre assessed a new training program de- 
signed to change personnel management prac- " 
tic<;s through a new performance appraisal sys- 
tem. The training program consisted of six nine- 
ty-minute sessions during which managers 
learned and practiced the procedures for the 
new performance appraisal system. The program 
concentrated on the application, of practices 
found in "good developers" such as extensive 
delegation and setting high standards. 

EVALUATION 

SeVen hundred managers in four distinct subor- 
ganizations of a large multi-corporation partici- 
pated in the study. Prior to training, subordinates 
of managers in all four groups completed a sev- 
enteen item questionnaire measuring those prac- 
tices stated as occurring in "good developers.*' 
Three of the groups were then trained while the 
foUrth group served as a control, maiiitaining the 
traditional trait approach to appraisal used by 
the company prior to the study. 
^The three trained groups were used to test the 
effectiveness of variations in the training pro- 
gram. The first group received training in the 
new system. They then were allowed to apply it 



on the job and were given feedback in the form 
of anonymous employee comments which were 
culled from the questionnaire which had been 
administered to subordinates. After the feedback 
session they received additional training. The 
second group received the same treatment as 
group one, but did not receive the additional 
training. The third group received no feedback, 
only the initial training in the use of the system. 

FINDINGS 

The greatest number of changes, as Goodacre 
expected, were recorded in the. first group. The 
\ control group^^ however, was a. close second due 
to an unplanned occurrence — increased - iinion 
pressure upon management to conduct perform- ^ 
ance appraisals and supply feedback annually, j 
Union membership in the control group was high | 
in comparison* to the three trained groups (72% 
against 19%, 16% and 23%). 
Even though the data showed that personnel 
nnianag6ment practices could be changed through 
, a program of training and new administrative 
procedures, Goodacre was not able to isolate the 
differential effects of the new program . because 
of the unplanned occurrences in the control 
group. 



Hand, Herbert H. and Slocum, John W. A longitudinal study of the eflfects of a human relations pro- 
gram on managerial effectiveness. Journal of Applied Psychology, 1972, 56(5), 412-417. . ^ . 

The author^ conducted this study to determine if a managerial human relations training course could 
change attitudes, and whether these new attitudes were reflected in organizational effectiveness. This 
article reports on the second phase of an evaluation two years after training. The first phase, done 
ninety days after trai/iing, reported no significant attitudinal or behavioral changes. 



TOAINING 

A packaged course was conducted by the Con- 
tinuing Education Division of a major -eastern 
university for a s;?3cialty « steel plant in central 
Pennsylvania. The training provided practical . 
application of human relations principles through 



problem solving methods. The "off the shelf " or 
standardized package program used, consisted of 
a total of nine hours of discussion of managerial 
styles and a total of thirty hours of experimental 
learning experiences using individual and group 
exercises, self assessment devices and ssmula- 



tions. A third segment of the course provided a 
total of three hours of instruction on cnrtent 
motivation theory. Training was accomplished 
during twenty-eight ninety minute sessions, be- 
ginning in September, 1968 and continuing 
through March, 1969. 

EVALUATION 

The evaluators, who were members of the uni- 
versity staff, used a pretest-posttest strategy to 
measure changes in behavior, attitudes and job 
performance. Twenty^one trained managers and - 
twenty-one controls were 'randomly selected from 
the same hierarchical level of the organization to 
participate in the study. A statistical comparison 
of corporate demographic factors, such as posi- 
tion in the hierarchy, time in present position 
and educational level, insured comparability of 
the two groups. 

Measurements were taken prior to training, nine- 
ty days after the experimental group had been 
trained and again fifteen months later. During 
each .testing, jnanagers in the experimental and 
control groups completed the Leadership Opin- 
ion Questionnaire, a measurement^ of leadership 
attitudes concerning work environment and task 
performance. Simultaneously, thSr subordinates 
received.the Supervisory Behavior Description, 
a companion instrument which provided subordi- 
nate measurements of the same leadership quali- 
ties. At the same time, their superiors completed 
a job .performance scale developed by, the corpo- 
^^ratiort's Director of Training. The traits rated by 
this instrument were technical knowledge, func- 



tional knowledge, drive/agressiveness, reliability, 
cooperation and organizing ability. 

HNDINGS ; 

The evaluators found that managers in the con- 
trol grpup described themselves as becoming 
more impersonal and production centered; their 
subordinates also described them as moving 
away from concern for human relations in the 
task environment. Evaluators noted a mirror 
image change in the trained group. Significant 
attitude and performance ch2;nges surfaced. 
Members of the trained group developed more 
positive attitudes toward the human relations 
aspect of their jobs, and these attitudes led to. 
positive changes in job performance. 

Since the researchers had anticipated significant, 
change in the experimental-" group only, they 
speculated, upon possible explanations for the 
unexpected change in the control group; A likely 
explanation was proposed. Noticing that the 
trained managers enjoyed greater .success with 
peers and subordinates and greater job effective^ ' 
ness, the control group managers reacted by 
applying a more autocratic, structured approach, 
one which they had found successful in* the firm >■ 
in the past. Increased managerial control could 
have in turn adversely effected subordinate per- 
formance, which was ultimately reflected ui de- 
creased managerial effectiveness: Without pre- 
and posttraining measures this phenomenon ^ 
might have remained unnoticed. 



Hillman, Harry. Measuring management training, a case study. Trainii^ Directors Journal, 1962, 16(3), 
.27-31.. ' ■ ■ ^ . ■ ' 



Hillman studied a training program designed to foster unifphn^^rsonnel policies within a large, rapidly 
expanding multiple organizatiooy and to improve the skills of its managers in interpersonal relajtions. 



TRAINING 

A training program was proposed to elinninate 
thie .widespriead inability of managers to manage. 
Through training, the org^ization hoped to re- 
duce , employee turnover iand absenteeism, im- 
prove operating (efficiency, improve safety and 
reduce Workmen's Compensation costs, improve 
employee morale. and^^'Sfitudes, and ret^d 
growth of union representation. 



The resulting three-day training program cov^ 
ered: management responsibilities for planning, 
organizing, communications, supervision and 
controlling; selection, indoctrination. and training 
of employees; . methods improvement; handling 
perspnnel problem cases; safety; and human re- 
lations. -A team of two headquarter managers, 
aiid a sefected local manager servdd as instruc- 
tors. Gourse developers employed the confer-' 



J? 



ence technique supplemented by actual case 
problems, role plays, visual aids and films. The ' 
tiainees were branch managers within the firm. 

EVALUATION 

To determine whether the goals of training had 
been met, the cvaluators employed a test .of 
classroom karning, measured trainee reactions to 
course and instructor, and assessed impact of 
training on the organization; '*How Supervise," 
a published questionnaire measuring changes in 
the participant's thinking about supervisory 
practices, policies and opinions, was adminis- 
tered at the begiiyiing and end of the course to 
obtain a measure of classroom learning. After 
each session, participants anonomously complet- 
ed a five-point rating scale measuring their reac- 
tions, to each conference leader's effectiveness 
and the interest value, usefulness and potential 
for application of each subject. Once the train- 
ees had returned to their jobs, monitoring of 
turnover, absenteeism^ accident frequency. 
Workmen's Compensation costs and uiiion gains 
provided a measure of the. impact of- training 
upon the organization. , " t 



FINDINGS 

The evaluators decided to discontinue the use of 
*'How Supervise" after consistently favorable 
results were . 6btained from nine consecutive 
course offerings. At that point they were satis- 
fied that the course was able to transmit learning 
in the classroom setting. 

.Overall, ratings obtained from the reaction scale 
w.^re generally high but conference leaders using 
the lecture approach received somewhat lower 
rating^. When asked for their views, however, 
participants expressed the view that basic quali- 
ties of cG.nmon sense and understanding of peo- 
ple could nqt .be taught by this type of training 
program. ^ 

The payoffs of this training program to the. orga- 
nization were high. Turnover dropped thirty per- 
cent, accidents decreased by fifty percent, and 
Workmen's Compensation costs were substan- 
tially lower. Union activity virtually stopped, an 
" outcome desired by the organization's top man- 
agement. 



Ivancevich;. John M. A ^study of a^cognitive training program: Trainer styles and group development. 
Academ y of Management Journai, 1974, 17(3), 428-439. ^ - 

Jvancevich examined the effects that varying the role^of the instructor had upon participant groups dur- 
ing training. ^ '■ ^ 



TRAINING 

A five-day management and organizationaK be- 
havior traiiiing program served as the stage for 
Ivancevich's experiment. The course was de- 
signed to teach conceptual knowledge to first 
level man'rSement within a multi-plant manufac- 
turing company. Text articles, foie plays ;^ a 
. communication exercise and questionnaire were 
used to teach motivation,^ group functioning; 
leadership, organizational change, behavioral 
science, organization,^ and company trends. Six- 
ty-four first level managers attended. 

EVALUATION ^ , 

Ivancevich explored the effect, of two contrast- 
ing trainer styles on group development. He 



hypothesized that he would find statistically sig- 
'nificant differences betvl^een the tv/o groups in 
group cohesion, interparticipant conflict, operwcss 
of communication, perceived interpretations of 
positive productivity and participant attitudes 
' to>yard training. ^ - 

With the help ^of the comijany's Vice President 
of '"Production, the EHrector of Industrial Rela- 
tions and the Assistant EHrector of Industrial 
Relations, Ivancevich assigned th^ sixty-four 
managers to two matched groups in which mem- 
bers were not well acquainted with each other. 
The managers were matched 'with Tespect to age, 
education level, tenure, and salary rates. - 

Both groups covered the same topics, received 
identical instructional and reading materials and 



had the same trainer. The dilference rested in 
the role played by the trainer. In one group, 
"training group structured," the trainer lectured 
at the beginning of each of the eight modules 
and specified what the learning expectations . 
were and how they would be accomplished. The 
trainer played an active and directive role. He 
provided reinforcement for good ideas and 
suggestions, gave immediate, constructive feed- 
back on completed tasks, and felt free to openly 
disagree with participants. The time spent on all 
activities was closely controlled. In the other 
group, "training group with minimum struc- 
ture,-' the trainer played a facilitative;role. He 
held lecturing to a minimum and presented his 
thoughts and ideas via the chalkboard. He ex- 
pressed opinions only when asked and avoided 
disagreements with participants. The group free- 
ly discussed whatever ideas and issues they 
wished to be clarified. The trainer provided a 



minimum of feedback and reinforcement, and. 
exerted no control . over the sequence of and 
time spent on instructional and reading materi- 
als. 

After each module participants completed a fifty 
item rating scale measuring attitudes toward 
group cohesion, interpardcipant conflict^ openess 
of communication, productivity of module ses- 
sion, and attitude towards trainer. The evaluator 
.traced, and examined changes in the development 
of both groups for each individual module. 

FINDINGS 

For the' most part, the structured training ap- 
proach was proven to be more effective. This 
group experienced more statistically significant 
irhprovements or group development in the areas 
of group cohesion,* openess of communication, 
productivity and attitude toward the trainer. . 



Leidecker, Joel K., and Hall, James L. The impact of management development programs on attitude 
formation. Personnel Journal, 1974, 53(7), 507-512. . ' 

Emulating an earlier study, the authors set out to measure the impact of an ongoing management de- 
velopment program on participants' attitudes and va/ues. Their results pointed to new directions for 
management development programs. , . 



TRAINING 

Leidecker and Hall studied a ten-week manage- 
ment development prograin at the Santa Clara 
Center for Leadership Development. The sub- 
jects of this study were drawn from the partici- 
pants in these programs beginning with the aca- 
demic year 1971-72 and continuing into the aca- 
demic year of 1972-73. The particii)ants, who 
were middle managers from diverse kinds of 
firms ixi the San Francisco Bay area, ranged in 
age from their twenties to their fifties, with the 
majority in their thirties and forties. ' 
«/ ■• 

EVALUATION . 

Through theu' research, the eyaliiators wanted 
both to measure the impact of . this program on 
participants- attitudes and values and to compare 
their findings to a similar study done seven years 
earlier by Schein.i Two instruments were used 
to collect the necessary data. The first, the Pub- 



lic Opinion Questionnaire, which consisted of a 
series of attitude "^nd value statements, deter- 
mined participants' positions on five dimensions: 
business and society, general cynicism, manage- 
ment theories and attitudes, attitudes towards 
people and groups, and individual and organiza- 
tional relationships. Participants completed it 
before and after training. A second instrument, 
ah adaption of the Profile of Oifganizational 
Characteristics, determined whether the partici- 
pant's organization was "highly supportive" or 
"lessv supportive." Administered before training 
only, its use was to gauge the relationsKip be- 
tween organizational, climate and permanence of 
participant's change^ if any. ' 

FmOINGS 

Managers in the present study ' overwhelmingly 
(90%) responded that ihey had^ faith in their 
workers. Oh this basis, the e valuators suggested 
that future training in the areas of Participative 



Management, Human Relations and Job Enrich- 
ment should concentrate on developing the skills 
of application rather than teaching the import- 
ance of these concepts. 

The evialuators also found strong support for the 
principle of supportive relationships. The pro- 
cesses within the organization were nO longer 
viewed as dehumanizing. Rather, the experi- 
emces provided by the organization were seen as 
supportive to the individual, and ones which 
build a sense of personal worth and importance. 
The strength of this commitment, lacking in a 
previous study ,2 indicated that managers had 
now adopted the necessary attitudes to change 
thCv^organizational relationships. The focus of 
trai'nfng^ programs should, therefore, shift from 



attitude chamge to concept application. 
Trje evaluators compared the present study's 
results with a similar one done by Schein in 
1967. Schein had studied a twelve-month man- 
agement education program conducted at M.I.T. 
The evaluators found that present participants 
possessed more positive attitudes toward corpo- 
rate responsibility than the earlier group. Fur- 
thermore, attitudes of present subjects shifted 
positively in this area, a change not noted in the 
Schein study. This implies greater receptivity of 
subjects to training programs emphasizing corpo- 
rate responsibility and social awareness. 

'Schein, Edgar H. Attitude change during management 
education. AdmlnlstraUve Science Quarterly, 1967, 11(4), 601- 
628. 

2|bid. 



McClelland, D. Achievement motivation can be developed. Harvard Businiess Review, 1965, 43(6); 6-24, 
178. ■ ^ --.v- - ' • 

McChnand^scribes and evaluates his approach to developing entrepreneurial traits in managers; 



TRAir«NG 

Mcfelelland reports on three studies of his own 
development program which was aimed at teach- 
ing participants how to improve their perform- 
ance as entrepreneurs. One study took place in 
the^U.S., the others . in India. In all programs, 
^participants were taught about McClellandls 
concept of need achievement (n-ach), its import- 
iance to entrepreneurs, and how to act and per- 
ceive the world like a .*'hi n-ach" (a person who 
IS highly motivated by the need to achieve suc- 
cess) 

He employs four main training method^tv (1) 
Goal Setting— trainees set goals which are then 
used as a target against which they can evaluate 
thei/ progress. Every six months for two years 
participants fill out a report form. This record 
keeping process keeps the goal salient. (2) Lan- 
guage of Achievement— the trainee learns to 
code his/her thoughts or fantasies and talk in 
terms of **hi n-ach' "concepts. This in turn influ- 
ences behavior. (3) Cognitive Supplorts — the per- 
son must re-examine pre-existing attitudes, val- 
ues and self-concepts and reconcile them with 
this new way of thinking and acting^ (4) Groiip 



Supports— instructors and peers provide emo- 
tional support. 

EVALUATION 

In order to determine whether , or not his course 
had been effective, McClelland compared parti- 
cipants' pretraining leyel of entrepreneurial ac- 
tivity to the level exhibited once they had re- 
turned to. their jobs. Increased cntreprenurial 
activity was defined as observable acts such as 
advancement within the organization, or ian in- 
crease In the size or scope of one's own busi- 
ness. 

APPLICATIONS 

McClelland describes three applications of his 
training program. The following b:ief summaries 
address the programs and their findings. 

Study I 

A pilot study conducted at a large U.S. corpora- 
tion involved sixteen executives in a one-week 
training session. Participants were matched to a 
control group on the basis of length, of service, 
age, job. type and salary level. Two years later 
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McClelland conducted a follow-up study lo find 
oiit which group of men had done better subse- 
quent to training. On the average, those who had 
attended McClelland' s course had been promot- 
ed faster than their matched controls. 

Study U 

The, second study, done in India, evaluated a 
similar training program conducted at the Small 
Industries Extension Training Institute. Fifty- 
two Indians from several communities attended 
the ten-day residential seminar. The men were 
heads of small businesses, lawyers, bankeri> and 
politicians. Trainees participated in individual 
V and group sessions, analyzed their self cd^epts 
based on psychological test data and learned the 
language of achievement (positive thinking) and 
how to set goals. A^follow-up period of six to 
ten months ensued. 



During the follow-up period* ^^t^ trainees dou- 
bled their entrepeneurial rat^- '^^fore training 
one third of the group had Unusually ac- 
tive, the number increased to two thrds with 
training. The judgment \vas b^sed on observable 
acts such as expanding a" ol^ business, increas- 
ing profit or investigating ^ n^^ Product line. 

Study HI 

A third study of thirty-two salaried executives 
from a variety of firmS in Bombay replicated 
these results. Before the course only about 
twenty to thirty percent of tH^ men were "quite 
active.^' A follow-up conducted t^o years after 
(raining revealed that t\VO thi^^^ ^^e nien who 
had attended the cours^ W^'"^ "unusually 
active."' 



ntal 



evaluation of 



Mahoney, Thomas A., Jerdee, Thomas H., and Korman. Abraham. An e^cperifTie! 
management training; P^rsonricl Fsychology, I960, 13(1), 81-98. 

Mahoney, Jerdee and Korman used a multhgroup experimental design to evaluate ^^<^ran effective- 
ness of a managerial training program and the relative effectiveness of two typ^s oi^^ studies. 



TRAINING . 

The goals of the on-going program were to de- 
velop within top and subordinate levels of man- 
agement a khowledgie and understanding of 
ina'^-agement principles, to develop the ability to^ 
appiy a speciial approach to problem-solving and 
to increase personal responsibility for self devel- , 
opment. The instruQtional" techniques used to 
achieve Jhese objectives were case analysis and 
group discussion supplemented by reading as- 
. signments and lectures. Two approaches to case 
analysis were used. Some, trainees received the 
standard case approach while others received 
the "boss involved" approach. The latter.ap- . 
proach requires air' particFpants to meet with 
their respective superiors prior to training in 
order to select a problem vcncountered by partici- 
pants in their work. The participant then analyz-' 
es the problem, prepares .a solution and reports 
back to the superior after training. The instruc-^ . 
tor selects one such problem for class discus- 
sion. The week long course took place at non- 
residential facilities near the work site. Trainees 



were sixty-four second l^Vel ni^nagers from one 
division of a large comp^^y' 

EVALUATION " - ^ 
To determine whether eithet* ^^f^ ^pproach was 
superior in achieving cct^se objectives the eval- 
uators cop:pared participants traiiied Using each 
case appioach to participant^ whov^ad not as yet 
been trained. The evalu^torS asSig-^aed the mana- 
gers ta-^ve groups, atte^ptiP^^^ Provide a mix 
of the dilferent functioii^l responsibilities within 
each group so that managers ^ould share experi- 
ences and learn froit» ofi^ another. Carefur 
scheduling of course sessions enabled the evalu- 
ators to have a control group- 

The first step was to t^st participants on three 
cnterion measures bef^^e ^"V !^aining began. 
: The "Management Pr^ctice^ Q"^^" >Vas devel- 
oped based on course materials to test knowl- 
edge of concepts and pdnciP^^^ Present^ in the 
course. A standard sc^^^ng P^'^^edure enabled 
the use of sample case Studies Assess problem 
solving ability. A third ni^^^^' an attitudinal 
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scale, was developed to measure responsibility 
for self development. 

After the pretest was completed, two instructors 
taught two sessions simultaneously. Instructor I 
taught Group I using the "bosg involved" ap- 
proach; Instructor II used the standard approach 
with Group II. Then the instructors taught two 
"»ore sessions simultaneously, jj^j^ Instruc- 
tor I used the standard case approach and In- 
structor II used the ' boss involved" . approach. 
Instructors I and II taught Groups III and IV, 
"■^spectively. ' """X, 

A posttest using the same three criterl&n mea- 
sures was then administered to al! five groups. 
.Group V, serving as control group, recelvey^ 
ti'aining after the experiment ended. \. 

FINDINGS '1 
^though the trainees showed no significant im- 
provement on the test of knowledge, significant 



improvement surfaced on the attitudinal tneasurs 
and^on one ^^^^^ of the case study. In seeking a 
possible explanation to the results, the evalua- 
tors compared the participants' scores on the 
criterion measures to those of the •instructors, 
who had attended an earlier course offering. In 
general, the instructors scored no higRgr after 
the Course than the participants had scored be- 
fore training, this suggests that either the in- 
structors Were not adequately prepared or the 
^criterion measures v/ere not relevant criteria for 
the evaluation of the course. 

Analysis also showed no difference between the 
"boss involved" approach and the traditional 
appi oach in the achieve'"ent of training objec- 
tives. No conclusion can be drawn, however, 
becau&p Participant comments indicated, that, in 
many instances, the procedures in the "boss 
involved" approach had not been followed 



■^aier, Norman R.F. An experit^gntal test of the sff^^^ °f i^^inine. on discussion leaderhip.. Human 
Relations. 1953. 6(2).,. 161-173. ^ ^ ^ 



{'^a/er set up an experiment to determine if supervisors could be trained to use a permissive approach 
'" iiiscussion leaderhip- 



TRAINING ^ 

An eight hour training course Was . designed to 
present the nature of group decision-making and 
it can be adapted to dealing ^th job prob- 
'«nis. The instniCtioiial methods used were lec- 
ture, group discussion, and roh playing. IXiring 
a four-hour discussion penod, trainees asked 
questions and expressed their opinions and atti- 
tudes. The role playing problem which followed 
allowed the trainees to apply thgi^ new knowl- 
*=dge and allowed the evaluator to test the train- 
ees' learning. In the problem, a foreman had to 
convince three employees to change their job> 
tasks." ' 

EV^UATION 

'^^determine whether the training program had 
produced a change in lear.n.ing, ^^^^^ compared 
trainee performance on the role piay exercisevto 
performance of untrained managers. Instead of 
training, managers in the <?ontrol group received . 



» one^-half hour lecture on- resistiince to change 
which served to introduce the role play, all, : 
forty-four groups of trained managers and thirty- 

. six groups of controls performed the role play. 
Jach of these groups Had four membe.s, one to 
play each role mentioned above. By using large 
. numbers of ^oie placing gro"Ps for l-.jth experi^ 
jnental and controf^^cionditions, Maicr hoped to 
overcome" Some of the bias which individual per- 
sonalities ititroduce into the role play setting By 
iiaving many groups the effects of individiial 

' personaii'ties cancel each other out. 

fINDlNGs . 

fhe trained leaders .^i^ere more effective in bring- 
ijig about Change than their untrained counter-, 
parts. ,_Kfty pgrcenT of the control group and 
^ty-nihe Percenf of the eXpenmental grpup "ac- 
cepted change" in the problem setting, pifty 
percent- of the control group ''rejected^change." 
in the experimentaf group, however, 36.4 per- 



cent accepted compromise and only 4.5 percent 
rejected change. 

A basic difference was found between the two 
groups. Untrained leaders attempted to force 
change by using money. Trained leaders viewed 
the situation as a problem io be solved by the 
group and tended to be more considerate^ toler- 



ant and open to suggestions. 

The evaluator concluded that the eight-hour 
course was sufficient to cause behavioral change 
in the classroom setting. However, higher man- 
agement support was. felt to be necessary if the 
new behaviors were to transfer to the job set- 
ting. 



Margulies,. Newton. The effects of an organizational sensitivity training program on a measure of self 
actua lization. Stud ies iri Personn el Ps ychology (Canada). 1973, S(2),, 67-74. . ^ 

T'he objective of this evaluation was to determine whether or not an extended sensitivity training pro- 
gram which provides the opportunity for reinforced practice^^yf learning increases th^ abiUty of the T- 

group ejigerience to develop self n^tualization in managers. 

.'t ■ 

TRAINING 

Margulies studied a T-group based: managerial 
training program conducted for middle managers 
in a large company. This particular sensitivity 
training program differed from the usua^ week- 
'ong approach in that training sessions extended 
over a four-month period and specific sessions 
Were devoted to the transfer of learning to the 
job. Margulies wanted to determine if this ap- 
proach could produce changes in the degree of 
self actualizaijon of participants. Fifty middle 
managers were selected to participate in the 
comparison on the basis of interest, ability to 
benefit from, the pro^ran) and lack of interfer- 
ence from job tension upon participation in the 
program. 

EVALUATION 

Margulies hypothesized that the provision of ex- 
pHcit application opportunities would be reflect- 
\ed in tHe^ psychological growth of the partici- 
pants. To test this, he exposed participants to 
different program variatipns, testing them* before 
and after training on a measure of self actualiza- 
tion. . 



To test his hypothesis, Margulies used a control- 
group pretest-posttest design. Participants were 
randomly assigned to five equal size groups. 
Four groups received training. Xhe fifth group, 
serving as a control group, was told that due to 
the large number of participants it was necessary 
to tficlude them in a sensitivity training program 
planned to be held several months later. 



Trainees attended a three-hour session each 
week and one full weekend session per month 
^over a four-month period. One of the trained 
groups was a "pure'' T-groupMn which instruo;^ 
tors mabe no attempt to link training to job e:^-'^^ 
periences. The other three trained groups^ had 
specific sessions \levoted to practice of specific' 
day to day "applications" and reinforcement of 
learning. 

The Personal Orientation Inventory, (POI) a 
measure of self actualization, was administered • 
to all five gi^oups in an orientation session *bef ore 
the program started, and again, once training, 
had ended. Tlie self actualizating person is one' 
who functions in the . present, becoming more 
aware of his or her own personal attrjt>utes.and 
potential, and better able to top that potential. 
The POI measured self actualizatioiTby assesing 
Inner Directedness, or 'whether the individual 
was motivated by self or other;, and Time Com- 
petence, of the individual's ability to function in 
the present rather than the past or the future, 

FINDINGS [ 

The three trained groups did significantly better ^ 
on the posttraining measure' than the untrained 
control. Groups had not differed significantly on * 
the pretraining measure. The results affirmed the 
value of training in increasing managers ability 
to self actualize. ■ 

Further analysis showed that the improvements 
;made by the ''applications" groiigs were signifi- 



cantly greater than those of the ''pure*' group. Directedness measure, while the ''applications'' 
The '*pure" group made gains only on the Inner group gained on b^th. ^ 



Mi rer, John B. Studies in Management Educati on. New York: Springer, 1%5. o 

Miner addressed the question of )^ether motives of management can be changed in the direction of i 
greater effectiveness and if this change coiitd be induced through training. 



TRAINING 

Miner presented his training course to research 
and development managers who were spending 
nearly all their time on research and develop- 
ment activities and little time in fulfilling their, 
managerial responsibilities. Initially the course 
concentrated on i^asons for ineffective sujbordi- 
nate performance, shifting in later sessions to the 
ineffective mianager. Reasons for ineffective per- 
• f onhance were presented in light of psychological 
and sociological theory with frequent referrals to 
research findings in those areas. 

Training was given during ten ninety-minute ses- 
sions, conducted at weekly intervrJs. MJner 
taught four groups, roughly, equivalem in size, 
..primarily^ using the lecture method v;dth some 
discussion when questions arose. Typically, 
small group discussion followed the fectures. 

EVALUATION 

Miner used a pretest-posttest control group design 
to^valuate the effects of training. He adminis- 
tered the Miner Sentence Completion Test 
(MCAT)^o seventy-two trainees and a contirol 
group of thirty managers in the same work unit 
who did not attend the. course. The MCAT uses 
the sentence completion format to measure 
managerial attitudes toward various aspects of 
the managerial role. Both groups filled out the 
MCAT during the first week of training arid 
again once training had been completed. 



FINDINGS 

On the posttest, fifty-nine percent of the experi- 
m'ental group inproved, twenty percent made no 
change^ and twenty-one percent dropped in 
score. Miner felt that this decrease was not due 
to the training, instead attributing it to external 
factors. A similar decrease in the control group 
supported his opinion that organizational change 
had been the cause. • 

Follow-up Study 

Miner conducted a follow-up using job related 
criteraa instead of the questionnaire. Fifty-two 
managers who took the course were compared 
to forty-nine untrained managers. Both groups 
were equivalent in salsiry and grade level at the 
time of training. Performance appraisal data in- 
dicated that both groups were faifly well 
matched. ' 

An examination of posttraining promotion rec- 
ords for those remaining with the compariy dur- 
ing follow-up, . showed the expierimental group 
signifkantly superior. Within the experimental 
group eighty-six percent had at least one promo- 
tion, while. only fifty-six percent of the control, 
gr9up experienced the same progression. Miner 
also examined rehiring recommendations for 
those who left the company during* the follow-up 
period. Only thirty percent'of the controls, as 
compared to sixty-nine percent. o^ the experi- 
mental subjects who. separated, did so with a 
statement in their files indicating that their supe- 
riors were sorry to let' them go. 



Moffie, Dannie J., Calhoon, Richard, and O'Brien, James. Evaluation of a management development 
: program. Personnel Psychology^ 1964,^17(4)/43 1-440. ♦ 

Moffie, Calhoon and O'Brien conducted a study demonstrating^ the need for more^han one measure of 
course effectiveness. ^ 



^TRAINING 

The consulting firm of Kepner-Tregoe and Asso- 
ciates inc. provided their probleim solving and 
decision-making course to the top three levels of 
managerial personnel of a large' southeastern 
papermilL The course lasted a total of twenty 
hours ^' 

EVALUATION 

The evaluators collected three distinct types of 
' data^ all of which were considered in the fin?! 
decision to retain the course: (1) a course^nd 
questionnaire\^ obtained trainee reactions to the 
program: trainees were asked for their opinions 
op the extent to which the course met its objec- 
tives, how the program -related to their job relat- 
ed problems an^ how the program might be im- 
proved; (2) observers recorded the behavior of 
^participants during practical work periods into 
pre-establisHfed categories in order to determine 
the effects of training on interpersonal relation- 
ships; (3) an experimental evaluation was con- 
ducted to determine the' effect of training on 



learning. Trainees and a second group of mana- 
gers from another mill, matched on the basis of 
biographical and psychological test data, re- 
ceived pre- and post course measures. These in- ^ 
eluded the Watson-Glaser Critical Thinking Ap- 
praisal, a standardized test measuring the ability 
to logically organize and analyze problems, and 
pre- and posttraining case problems developed .^.^ 
by the consulting firm. 

FINDINGS 

The evaluators found no statistically significant 
difference between the trained group and the 
control group on the experimental measures. 
They found, however,, that the trainees thought 
the program to be worthwhile and were applying^ 
the techniques taught in the course to day to day 
manufacturing problems. They concluded by 
reporting that ''Management plans to continue 
use of the training program in view of the fact 
that in specific situations the techniques learned 
inihe course appear* to be beneficial in the solu- 
tion of every day manufacturing problems." u 



Moon, CO., and Hari ton, Theodore. Evaluating an appraisal and feedback training program. Person- 
neU 1958,35(3), 36-41. . . ■ 

Moon and Hariton demonstrate that meaningful results can be obtained from relatively simple measur- 
ing methods. - * • 



TRAINING 

In the spring of 1956,/the Engineering Section of 
a department of the General Electric Company 
installpd a new appraisal and personnel develop- 
ment program. The new program featured a re- 
vised performance appraisal system and a thirty- 
hour course teaching line managers to use ap- 
praisal information in helping subordinates devel- 
op themselves. The program stressed both the 
essential principles and interviewing skills neces- 
sary to feed back appraisal results in a manner 
designed to stimulate employee growth and de- 



velopment. Fifty niianagers attended half-day 
sessions conducted in-house for a period pf two 
weeks. During these sessions, they received lec- 
tures and participated in a discussion and a role 
play. 

EVALUATION 

Two years after the adoption of the new apprais- 
al system a decision was made to evaluate its 
effectiveness. A consultant ffom the, Educational 
Testing Service and an in-house evaluator joined 
forces to determine the effects of a new apprais- 



17 




al and a personnel development program. Ques- 
tionnaires were sent to sixty-six raridomly select- 
ed employees in the Engineering Section and a 
cmlrol group of sixty-seven employees in the 
company's Manufacturing Section. The latter 
group had used a different appraisal system and 
their managers had received no training in feed- 
back techniques. The questionnaire asked subor- 
dinates for theii opinions concerning changes in 
their superior*s job behavior and attitudes. Ten 
dimensions of effective supervision were as- 
sessed e-y the posttraining measure. Managers 
who attended ti'aining also received a question- 
naire at this time asking their views about how 
they themselves had changed. 

FINDLNGS 

Subordinates in both the experimental and the 
control groups expressed that an overall im- 
provement in their managers' behavior and atti- 
tudes had occurred as compared to two years 



ago. The experimental group perceived their 
superiors dgnificantly higher than the controls 
on eight out of ten dimensions, the greatest 
differences in perceptions fell in the areas of giv- 
ing recognition for good work, providing oppor- 
tunity for subordinates to express their view- 
points, and expressing concern for their employ- 
ees', futures - gains clearly attributable to the 
training. In addition, the course paiiicipairts re- 
ported that they felt it easier to talk to- their 
employees abput job problems, and were able to* 
suggest solutions. On the other hand, the majori- 
ty felt that subordinates did not share any more 
of their job problems, with them than they had 
previously, nor did they feel that working rela- 
tjfinships^with peers had improved. 
Tne evalu^tors concluded, based, on this study, 
that the course definftely had a positiVv^ impact 
on the engineering managers and their subordi- 
nates, although there were other , factors, not 
examined in the study, which might also, have 
had an effect. 



Schein, Virginia. An evaluation of a long term management training program. Training and Dev^iopr 
mcnt Journal, 1971 , 24(12), 28-34. . ^ . 

Schein describes a study which attempted to assess the impact of a long-term training program upon 
participant interests, attitudes and personality characteristics, and to determine if predictors of these ^ 
changes could be found. ^ ^ 



TRAINING 

Sqhein examined an eight-month management 
training program conducted for college graduates 
by the American Management Asso<^iation. It 
had three objectives: (1) to prepare young peo- 
ple in philosophies, principles, skills and actual 
tools of management; (2) to make/participants 
highly desirable candidates for employment in 
any kind of organization they wish to enter; and 
(3) to !jclp participants gain intellectual maturity, 
emcUonal stability, and leadership capability 
more^ quickly than could be attained through ' 
other sources. ' 

The course utilized instruction^ methods- such 
as lectures, group discussion and films as well as 
programmed instruction, video tape business' 
simulations and T-group sessions. Instructors 
were drawn from the ranks of practicing mana- 



gers', consultants,,^ labor leaders, government 
officials and educators to. instruct in .their respec- 
tive areas of expertise. iTie trjunlees included in 
this study were 124 male members of the first 
four course offerings (1968-1969 term). All were 
college graduates and ranged in age from 21-31. 

EVALUATION -'y 
The purposes of the evaluation were twofold: (1) 
to determine how the learning experience effect- 
ed changes in participants' interests, attitudes 
and personality characteristics, and (2) to deter- 
mine- if predictors of these ' changes could be 
found. During" the first one. and one-half weeks 
and again, eight months later, just prior to gradu- 
ation, participants received a battery of • varied 
assessment instruments. These ini^truments were 
selected to assess changes occurnng in the parti- 
cipants attributable to training. Included were: 



the Study* of Values; Strong Vocational Interest 
Blank; "Guilford-Zimmerman Teniperament Sui': 
vcy; Leadership Opinion Questionnaire; Rotter's 
Internal-External Scale; and three measures de- 
sigf)^ tft measure attitudes toward maniage-, 
ment-^Influence Questionnaire, Public Opinion 
Qvr .onnaire and a portion of the Participant 
I jrmalioh StiayeyT 

Further measures were administered only once 
during the first week to determine whether or. 
not they could serve as potential predictors of 
change. These included: Adaptability Test; Otis 
Self Administering Test of Mental Ability; Miller 
Analogies Test; California Te?t of Personality; 
Edwards Personal Preference Schedule; Gordon 
Personal Inventory and Profije;- Biographical 
Invento/y; ^nd a portion of the Participant Ihf or- 
iliatipn Survey dealing with previous experience. 

FDrt«NGS 

The evaluators compared pretest scores with 
pbstte^ scores on all tests and subtests of tne 



battery. Significant change occurred on thirty- 
two of the seventy-four subtests administered, 
confirniing the effectiveness of the course.- 
Changeis fell into five general areas: leadership 
styip, attitude toward business, self confidence, 
need for sociability and. diversity. These changes 
seemed to correspond roughly with the objec-. 

tives of the program. 

• ■' \. ■ 

After differences in pre- and posttest scores 
were calculated, the potential predictors 'of 
change were compared to (correlated >yith). those, 
measures of change which showed significant 
differences between pre- and posttest scores. 
Results revealed that predictors of these changes 
could be identified. Biographical indicators of 
change surpassed intelligence and personality 
factors in their ability to predict change. These 
findings suggest that measures of . background . 
and past experience could be used to identify 
those individuals most likely to benefit from this 
program. 



Schwarz, Fred G., Stilwell," William P., and Scanlon, Burt K. Effects of management development on 
subordinate behavior. Tralnii^ and De velopriient-Joumal, 1968, 22(4), 38-50; and 22(5), 27-30. 

The evaluators demonstrate how a time lag design can be used to obtain a control group when ope 
would ordinarily not be available. Tbey also point out the difficulties of this design. ^ 



TRAINING 

A large midwestern insurance corporation devel- 
oped a. university based general management 
$eqiinar. in' conjunction with, the Universiity of 
Wisconsin. The course had twa units. The first, 
**Effective Executive Supervision," stressed^ 
dealings with people- The second, **TKe- Man- 
agement Process," dealt with work aspects such 
as planning, organizing, controlling and . delega-; 
tion. Each session lasted^ three days. Professors 
from the^ University of Wisconsin and other 
universities, business leaders, and consultants 
instructed. Workshops in which participants 
could'test^but learning ^yere ' stressed. Other in- 
• structional methdds^ employed were lectures,v 
directed discussions, case, studies, buzz groups i i 
films, and role plays. Fifty-^even managers rep- 
resenting five levels of top management attend- 

ed. ' ; ;# -' : ^'^-^v. " T . 



EVALUATION 

An evaltJation study was designed by university 
personnel to assess the impact of the course 
upon management behavior and to identify sub- 
ordinate reaction to any change. The' evaluators 

. employed; a special case of the pretest-pogttest 
control groiip design called the cycle" or time- 
lag design. As in the former design, the prospec- 
tive^trainees are divided 'into tyio groups — : the 
experimental group and .the control group. How- 
ever, in the^timerlag design both groups ultimate- 

' iy . receive training. First both groups take the 
pretest, then one -group is trained, Then, both 
i^oups tajke the posttest.; F'inally, the isecond 

" ^bup receives .training and further comparison 

:<~^^easures may be madei This "procedure allows 
' tor the establishment of a;^control group without 
depriving anyone;^of training, . ' 



The prospective participants in this study were 
random^ divided into two groups and then 
trained six months apart. Group ''A" started 
training in September .1965, and Group "B'\ in 
Fejjruary 1966. 

Between the two sessions, former and prospec- 
tive participants were interviewed ahd asked to 
describe critical incidents in their jobs and how 
they handled them. No mention of, or connec- 
tion to the university program was made. The 
critical incident portion of the evaluation mea- 
surement assessed the impa^lopf training upon 
managers^' behavior. — 

. Before and after their managers were trained, 
subordinates completed the Leadership Behavior 
Descriptipn Questionnaire (LBDQ). The instru- 
ment describes the leadership practices or activi- 
ties of the superior as perceived by the subordi- 
nate without assigning value judgments to the 
behavior. This provided an overall measure of 
training effectiveness. ' ^ 

FDJDINGS ' 

The following conclusions were derived from 
interview results. Group "A" showed a greater 
tendency to emphasize personnel events over 
production related events. They also placed 
greater emphasis on employed attitudes and had 
a greater concern for employee development. 
However, when control Group "B" mentioned 



personnel during interviews they spoke maiply 
about staffing and resolving conflicts, and had a 
great concern for production. The groups also 
differed in their methods and effectiveness of 
problem solving. Gfoup "A" reported greater 
effectiveness through the use of higher levels of 
motivation, employee involvement and develop-* 
ment; Group "B" used persuasion and sales- 
manship to, solve problems. Group "A" also 
displayed more of a tendency to learn from ev- 
eryday experiences. 

-* 

A second comparison between the two groups 
was made ba^ed on the results of the subordi- 
nate questionnaire. .After training. Group "i^'* 
was perceived as becoming less active, less de- 
finitive and less production-centered. Subordi- 
nates described the Group "B*' managers as 
moving more toward structure and becoming 
mdre active. The course, therefore, had opposite 
effects ^on the two groups. 

The evaluators suggested that these differences 
were due to the time gap, rather than the train- 
ing pro-am. Many members of both groups had 
daily contact with one another. Because of the 
interdependency^* of people working within the 
organization^ the time lag design allowed the 
"negative" results of Group "A"/s training 
experience to influence the outcomes of Group 
"BV's experience. Tlie evaluators were, there- 
fore, unable to reach any definite' conclusions 
based on the results of this study. 



Thorley, S. Evaluating an in-company management training program. Trair'ng apd Development Jour- 
nal, -1969, 23(9), 48-50. 



This study examines an attempt to assess training related behavioral changes in the applied work set- 
ting. 



^AAJNING " • ^ 

Tfiorley describes the evaluation of a training 
course given to 234 managers .working in a large 
English telecommunications company. The one 
week course was diesigned to broaden the knowl- 
edge, of the managers, expose them to new 
ideas, and to create the sense of functional inter- 
dependence. Each day, managers, received a 
different topic. Topics includecTihariceting, tech- 
nical manufacturihg, finance, and^rsonnel. 



Course speakers, for the most part, came from 
areas of specialization within the company. The 
training program tcfpk place in the company's 
small training center in London. 

EVALUATION 

The e valuator devised a questionnaire to collect 
from the participants possible criteria outside the , 
training .vsitiiation against which course success 
could be measured. The questionnaire was com- 



pleted aiionomously by participants at the close 
of the course. From the responses, the evaluator 
derived six categories: introduction of^ new tech- 
niques, financial control, cooperation with other 
functions, attitude toward job and company, 
staff development and decision making. To as- 
sess whether changes in behavior, skills or atti- 
tude had resulted from training, interviews with 
each trainee's immediate superior were arranged 
six months later. Each superior was given a list 
of his subordinates who had attended the course 
and asked about the performance of the group in 
each of the six areas. 

FINDINGS \ 

One half the superiors cited xhange in the spe- 
cific areas of financial control, cooperation with 



other functions, attitude toward job and comp- 
any and decision making. The other half indicated 
positive, changes had occurred in these areas, 
but could not be specific. One third^ reported 
change in staff development; a quarter reported 
introduction of new techniques and a quarter 
perceived a change but couldn't be specific. 

Thorley noted that there might have bggn some 
imprecision in the measurement techniqu^Hised 
in this stiidy, but he believed this appfpach to re- 
preferable to course ratings, speaker ratings and 
post courses reports. It may .mean more work 
for the training manager, but the results obt£iined 
through this approach are directly related to the 
course objectives which had been derived from 
a perception of training needs. 



Valiquet, Michael I. -individual change in a management development, program. Journal of Applied Be- 
havioral Science, 1968, 4(3>, 313-325. . • • - \ \ 

Valiquet examined participants on-the-job behavior to determine wbetherjearning from an experimen- 
tal organizational development program had resulted in changed attitudes and behavior. 



TRAINING 

The training program uAder observation was an 
on-going lab-type organizational development 
program with provision for guided on-the-job 
^application of learning. The primary goal of the 
program was to optimize the company's use of 
its human resources and to improve collabora- 
tion in working toward common gosAs, To this 
end,, participants were taught to analyze the con- 
sequences of their own actions and encouraged 
to (.consider and experiment with alternative 
management assumptions, attitudes, and behav- 
ioral patterns. • 

• . • • '""J . * . ' ■ 
J. . 

EVALUATION f , 

About one* year. after lall training had been c'om- 
pleted, Valiquet underjtoel^4he task of evaluating 
the training effort. He sought to ag sfess the im- 
pact of the program on the attifuaes and behav- 
ior of the participants in terms of the stated 
training objectives. He did this by comparing 
data from trained and untrained managers for 
evidence of such changes. ^; 



Valiquet selected at random sixty participants 
from four different company locations. He se- 
lected these four because in each one, the train- 
ing had begun at least one year prior to the eval- 
•"uatiori* Trainees therefore had ample time to in- 
ternalize the new values, attitudes and behavior- 
al skills in the working environment. Also, the 
researcher could determine how much learning 
had "survived the state, of early post-training 
euphoria" and the pressures and -adversities of 
the work situation, l A "matched pair control 
group" \vas obtained by asking each experimen- 
tal subject to nominate another manager who 
held an identical or almost similar functional role 
to the subject, but who had not participated in 
training. v 

Evaluative data was provided by "describers" 
^ho were individuals who had worked with the 
subject for the past year. They were selected 
from a mixed list of seven to ten peers, superi- 
ors and subordinates, submitted by each partici- 
pant. Five describers selected randomly from 
each participant's list received an dpen-ended 
questionnaire asking the individual to describe 



any specific ch^x\0^ tftat had occurred since the 
J, previous year, ^yt^jec^^/^^^ive^ ^ similar ques- 
tionnaire to di^\i des^'*^P^*^^s of their own be- 
havior. ' /' \ 

Describers' resj^^fi^es categorized accord- ' 

ing to tv^enty^ne content sub-categories 
grouped into th^^^ major categories, found to be 
organizationally ^(i^ personally relevant in two 
earlief studies doj^e bjf iles2 and Bunker3. Due 
to poor overall ^^turri rates, final results were 
available for Ox^iy tifirty-fouT trained subjects 
and fifteen con t^^j^. 

FBVDINGS 

StatisticaUy sigiy^c^t differences between the 
experimental an^ Control group describers were 



obtained on **total number of changes ob- 
served", **total changes agreed upon by two or 
more observers", and total number of changes 
reported by the subjects themselves". Valiquet 
attributes the strong positive results to^the fact 
that thie program involved inhouse training, con- 
ducted with co-workers, by company trainers. 
These factors facilitated the transfer of actual 
behavior to the work situation. 



•Miles. M.B. Processes and outcome. Journal of Counsel- 
ing Psychology, i960, 7(4). 301-306. 

21bid. ■ - . 

3Bunker, D.R. Individual applications of laboratory train- 
ing. Journal of Applied Behavfora} Science, 1965, 1(2);. 131- 
148. 



Vogels, Captain p^Vid Evaluation of a management training course. Journal of the American Society 
of Training Pire^^rs^ 1958, l2(l)M4^51. 



Vogels'used flU^^ffoniiBires to test the effectiveness 
ment program^ 

, ' . i 
TRAINING 

Vogels' describe^ ^ rtianagement training program 
for first and scq^^c} i^vel supervisors in the Air 
Force. It was d^gj^ned improve their supervi- . 
sory' skills and ^^tabli^h tWo-way communica- 
tions within t^^^ Qxg^f\\z^Won* The training 
course covered ^j^^cifjc aspects of the supervi- 
sors' job, work gitxiatjon iniprovement, produc- v 
tivity, problem ^^^IvinS' interpersonal relations, 
improvement of ^^if^ ?nd^ employee develop- 
ment. Over a ot^^ period trainees attended 
twenty-five twox^^t conferences conducted dur- 
ing weekly two^^our sessions- The . **develop- 
mental approacl^M ^ a modified case study ap- 
proach, and woj^K^hops were the dominant jp. 
structional meth^^?. 

After each conf^j-^nc^, trainees compiled a list . 
of job related p^oflenJS and forwarded it to the 
course monitors, fhes^ problems Were then dis- 
cussed with trai^^^sv i"™^diate ^pervijsofs at 
monthly meetii)^^ held by the evaluator and 
management anq ^jh^ng^s were agreed upon. In- 

.frequent meetin^^ AVefC held with superiors two 
levels above th^ (f^in^^s to discuss unresolved 

, problems, thes^ j-^Sulfs were, jn turn, fed back 



of a combined training and organizational develop- 



to the trainees' immediate superiors. Through 
this circuit, the ^valuator hbped to help establish . 
organizational commitment' and open up two- 
way lines of communication throughout orjgani- 
zational levels. 

EVALUATION 

To test the effectiveness of the combined train- 
ing arid organisational development program, the 
evaluator examined trainees' and immediate 
supervisors' perceptions while training was still 
in progress, \ questionnaire administered to 
trainees asked them how' effective they thought 
the course was and asked them if they perceived 
change in their behavior at the point of question- 
naire dissemination. A similar questionnaire giv- 
en to immediate superiors during . one of ^their 
feedback sessions asked for their feelingy about 
the effectiveness of the training program feed- 
back sessions. 

FINDINGS 

The questionnaires showed that while both the . 
trainees and their supervisors were satisfied with 
the training, the trainees were not cognizant of 
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'their superiors' satisfaction. The feedback tech- 
nique has only succeeded in opening upward 
^^lines of comiriunication, not in facilitating the 
downward flow. The course nonetheless was 



found to meet its objectives by improving super- 
visory skills. It >yas also responsible for the in- 
troduction of various management improve- 
ments:. ^ . . 



li 



i^Wexlgy, Kenneth N. and Nemeroff, Wayne F. Effectiveness of positive reinforcements and goals set- 
vting as methods of management development. Journal of Applied Psychology, 1975, 60(4), 446-450. 



Tfie evaluators employ an experimental design to compare the effectiveness of two variations of a su- 
pervisory skills training program, ' 



TRAINING 

The objectives of the training program were to 
develop a moire considerate, democratic style of 
leadership of managers working, in a large urban 
medical center. It was hoped that through the 
training of managers" subordinate job satisfaction 
could be increased and absenteeism reduced. 
The training program, had two phases. . Phase 1 
cQnsisted of two half-day workshops given , on 
consecutive days. Instructors relied on role play 
exercises to teacH those behaviors aj^propriate to . 
democratic, considerate supervisors. ''Tfainers-*^.^ 
provided specific feedback to trainees about the 
effectiveness of their performance. Phase II in- 
volved guided application of these techniques on 
the job. • - 

Nine departments of the medical center were 
selected to participate in the. program. All de- 
partment heads were put through the training 
exercises* in order to insure their understanding, 
commitment and support of the program. Twen- 
ty-seven of their imftiediate subordinates, three, 
from each department, then participated in vari- 
- ations of the two phase program. 

EVALUATION 

Wexley and Nemeroff compared three different 
variations in the training program to determine 
which of these had the greatest effect on subor- 
dinate perceptions (behavior) and absenteeism 
(results) vwthin the organization. 

The participating managers were randomly as- 
signed by department to one of two experimental 
groups or the control group, such that each de- 
partment had' the same nurpber of managers par- 
ticipating in each group. , 



Experimental Group I received the following^ 
treatment or training sequence- Before each of" 
the several role plays, the trainer discussed with 
the trainees a list of effective and ineffectiye 
behaviors specific to the particular exercise. 
Instructional staff observed the trainees' partici- 
pation and then provided a * 'delayed appraisal"* 
which incorporated specific feedback with rein- 
forcement. Trainees were told how well they 
were doing, and how much improvement they 
had made. During the second phase, lasting for 
six weeHs back on the job, trainees filled out a 
behavioral checklist daily, to keep a record 
their performance. The trainer interviewed train- 
ees at the end of the first and third weeks to re- 
view checked behaviors and to provide consulta- 
tion on any particular problem trainees had in 
the application '^of neW behaviors. During the 
' first meeting, trainees set goals for review at the 
second meeting. The trainer encouraged theni to 
set as their goals, the application of behaviors 
not previously checked. . ^ 

Experimental Group II received the same treat- 
ment with the addition of "telecoaching," or 
direct reinforcement given during exercises 
through a device worn on the ear of the trainee. 
The reinforcement was immediate in nature and 
behavioral ! n context. ' * 

The control group received neither **telecoach- 
ing" nor the behavioral checklist and did hot set 
goals for review, but otherwise experienced the 
same training sequence. 

Sixty days' after completion of the training the 
evaluators collected , measures of managerial 
behavior ^and job satisfaction from subordinates 
of the trainees. Between three' and six of each 



managa-'s subordinates were randomly selected 
to provide information on their superior's behav- 
ior on the .Leadership Behavior Description 
Questionnaire and two scales .of job satisfaction. 
Absenteeism records were also examined at that 
time. - *■ , 

FINDINGS , 

Members of both experimental groups improved 
slightly more than the control group members 
did. The improvement was measured in terms of 
subordinates' perceptions of managerial behavior 
and of absenteeism within the .organization. Im- 
provements were ' accomplished without any 
undesired . detrimental effects to the managers' 



o ability to direct the group or emphasize produc- 
tion goals. The basic treatment provided to 
Group I was significantly more effective in in- 
creasing subordinate satisfaction than the^ treat-, 
ment given Group II, featuring ''telecoaching" 
via the ear device. 

Post experimental interviews revealed com- 
plaints that the ear apparatus had disrupted the 
conversational flow of the role plays. In addition- 
it contradicted the participative management 
style preached during training, since it only al- 
lowed for one-way communication. 



Wilson, John, E. Mullen, Donald P., and Morton, Robert. Sensitivity training for individual. growth — 
team training for organizational development Training and Development Journal, 1968, 22(1), 47-53. 

Two different approaches to sensitivity training are compared to determine their relative effectiveness 
within ^the organizational setting. .. ' " 



TRAINING 

The evaluators studied two offerings -of an inter- 
agency management development course provid- 
ed by the training Division of the State Person- 
nel Board of the State of California for upper 
level managers in State Service. Consultants. 
Morton and Wilson conducted both offerings. 
The six-day residential course emphasized inter- 
personal relationships and leadership" skills. The 
first program, conducted for forty participants, 
in November, 1964, used the traditional sensitiv- 
ity approach. The second, conducted one year 
later for forty-one participants, used a modified 
versfon of the Organizational Training Laborato- 
ry, an approach emphasizing team . decision- 
making and problem-solving. 

EVALUATION / . - 

Mullen, a:management analyst with-the State of 
California, conducted the evaluation. He de- 



signed a questionnaire to determine the mana- 
gers' perceptions of the value of their training 
-experience to themselves as individuals and as 
managers in their organizations. The question- 
naire included specific questions, with scaled re- 
sponses and an open-ended question asking man- 
agers to cite things they were now doirig that 
they hadn't done prior to the course, both as 
individuals and as managers within the organize- 
tion. The questionnaires were sent out six 
months after the second group of managers had 
been trained. ' 7 . 



FINDINGS 

The authors found both courses to be of equal 
value to the managers as individuals. ^The Orga- 
nizational Training Lab, however, proved superi- 
or for improvements in team work and better for 
inteipersonal skills and problem solving. 
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GLOSSARY 



Battery — A series of related tests administered together. 

Behavior — An aissessment of individual performance on thie job. (Se i\ Data.) 

Client — ^The person(s) or organization(s) for which an evaluation is conducted. 

Control Group — A group, of persons who do not participate in the training program but .who are similar 
in all relevant respects to those who do participate. Control groups are used as a basis for compar- 
ison. . 

Correlation — A statistical process which shows the degree to which two or more events or objects are 
related to each other. In evaluations of training program .effectiveness, correlations may be calcu- 

,lated to determine the relationships' among factors effecting training results. 

.. ■ ' -\ 

Criterion, Criteria — Measures of training effectiveness which reflect the goals and objectives of the 
training program.,: They provide a description or image of what should ^happen, thereby facilitating 
. comparisons between jvhat should Have happened and what did happen. 

Data — Factual material from which conclusions may be drawn. When evaluating training, four catego- 
ries of data may be obtained: Reaction, Learning, Behavior and Results. 

Design — A strategy which the evaluator uses to collect data. The design usually specifies who wijl be 
measured (experimental group, control group), and when they will be measured (pretest, posttest). 
The purpose of the design is to guard against the possibility that something other than the treat- 
ment causes the observed effects of the training program. , ; 

Evaluation — A deliberate process which provides specific reliable information about a selected topic, 
problem, or question for purposes of determining value and/or making decisions, 

Experimental Design— A data collection strategy which attempts to control as many relevant and irrele- 
vant variables as possible. An experimental design gains its rigor by using contsxil groups and ran- 
j^om selection and assignment of individuals to groups. , 

Experimental Gjoiip— A'^group of subjects who receive the experimental treatment in a 'design. 

Forced Choice — A special kind of multiple-choice item which forces the respondent to choose the more 
"descriptive of two or more equally attractive or un-attractive statements. This type of item is more 
often used in personality and attitudinal measures when social desirability of the possible respon- 
ses may interfere with the selection of the alternative chosen. 

Formative Evaluation — ^The process of judging an instructional package or. process or its components 
during the developmental period for the purpose of providing persons directly involved with the 
formation of the entity with feed-back as to possible improvements. (See Evaluation.) • 

Historial Data — Data collected by the organization as part of its normal day-to-day functioning. It can 
include numerical -indices such as absenteeism, turnover or production rates, and organizational 
documents such as agency. memoranda, auditing reports, program budgets, employee rating forms, 
supervisor appraisals and written plans. -J: 

Hypothesis^A statement proposing a plausible relationship between two or more variables. 

Learning — ^The principles, facts, skills and attitudes that participants gain from. training. (See Data.) 

Matching — A pairing of subjects on the basis of background information factors such as age, level of 
' education or organizational status, followed, by random assignment of one member of thie pair to 
the experimental group and the other to the control group. This process, used >yhen totally random 
selection and, assignment are impossible, helps prevent the personal characteristics of the subjects 
from contaminating the evaluation results. 



Open-ended — A question allowing respondents to^'answer freely in their own words rather than restrict- 
ing their answers to a few stated alternatives as in a multiple-choice question. Although they are 
more difficult to analyze than multiple-choice qUiestions, open-ended questions allow for a wider 
variety of responses. ' . 

Posttest, Postcourse or Posttraining Measurement — A measurement taken after the training program has 
ended. The resulting information can be used to determine >vhether or not trainees have achieved 
training objectives. If compared to a pretest the posttest provides a measure of change probably 
attributable to training. 

Pretest, Precourse or Pretraining Measurement — A measurement taken before the training program be- 
gins, or^,during its early stages. The resulting information may be used to provide instructional de- 
signers with a picture of the skills and abilities of the average entering trainee, or may be used to 
give the instructor an idea of how much the group already knows relative fo the learning objec- 
tives. A pretest also pn>vides a baseline for comparison against a posttest. , 

Procedures — Instruments^^devices used to obtain data for evaluation. ^ \, 

Random Assignment or Selection— The selection of cases or subjects in such a way that all have an 
equal probability of being included, and the selection of one subject has no influence on the selec- 
tion of any other subject. 

Rating--The process of judging someone or something according to predetermined criteria. (See scale). 

Reaction— An indication of how well the trainees liked a particular program, including materials, instruc- 
tors, facilities, methodology, content, etc. (See Data.) 

Reliability— The degree to \vhich a device or instrument (procedure) measures a given characteristic 
consistently. ^ 

Results— The impact of training on the organization or job environment. (See Data.l 

.*-',. ... . " . 

Rigor— A term used to describe the amount of control exerted by a design and the consequent preci- 
sion that can be attributed to the findings. The more rigorous the design, the greater is the level of 
confidence that one can have in the findings of apL^evaluatibn. ; :- 

-Sample— A subset of the population, usually selected to be representative of the whole group Being 
studied. / ' . 

Scale— A graduated continuum which allows a rater to assign numerical values ranging from low to 
high to a given trait or characteristic. Scales generally have betweeri'three and nine categories 
which may or may not have accompanying descriptive adjectives or numbers. 

Significant, StatisticaUy Signiticant-^A statistically significant event .is one that has a low likelihood^ pf 
happenmg by chance. Significance does not mean importance; it merely means that a difference, 
such as the difference between the scores of two groups on a posttest, was due to some difference 
between the two groups rather than due to chance: . / 

Subjects— Individuals selected to participate in any facet of a design. 

Summative Evaluation— The process of judging a completed instructional package or process for the 
. purpose; of providing the end user with information as to its demonsttated effectiveness in a given 
situation. Based on this information, the user may decide to purchase the entity (if a potentiaf 
user), or retain it (if a current user). (See Evaluation.) > 

Treatment— The training program or a variation in the training program given to an experimeiital group 
; in a design, . . ^ 

Test— A series of questions, exercises or other means of measuring the knowledge, skills, abilities or 
aptitudes of an individual or group against selected criteria or norms. 



Validity— The degree to which a device or instrument (procedure) measures; what it was intended to 
measure. . - 

Variable — Something thai is capable of ihariging in value over tfme. One purpose of a design is to con- 
trol for (limit the variability oO irrelevant variables so that the effect of relevant variables may be 
observed. ' 
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